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IN A LARGE URBAN SCHOOq SYSTEM 

By ■ ' 



° Midiae^i H. Kean 



lntroductiorv \ i^ , » { *" • , v ' ' • . 

Rapid, .^ad^pften uncontrolled growth has/-character *rzed ^urban^ sch^or 

: c • U ' \ * • . I ^< ^ ^- ' . T 

district-based research and e\/;a 1 uat iofi Viepartmen*ts since the advent pf 
the^EJem^ntary "and ^econdary Education Acj: of I965. 'The^viev; ifhat • 
evaluation,' a^ the proces^ of providing information for decision Quakers, 
is vital to effective school "^system admin istrafi on has developed 
*simul taneous ly, with increasing Federal involvement, in education, 
particularly, in urban areas. Although. the overal 1. refcogni tion and 

•1/ ■ y ' 

expansion ofpr-the roles of^educational researchi and evaluation -Should be 
^viewed positively, such raj^id development does not take place wiyiout 
.complications. • •* / ' J , . 

^The ^jpj^V\y^% of thi^ paper relate to' the identification and 



-0* 



examination of tFiose areas mos^t critical tQ the effeo^ve managenpjit of 
a researich and ievaJ-uation 'department in a large urban school syst,em. , 
Th^ paper >/i 1 1 focus upon** the organisational structure and management 

*r * * V ' • ^ ' i* . ; t 

cc/ricerns of sucfi opera^tions, and cite actual examples of how .the latter 

are being/dea-^t with. 

' (• \ , ^ • ' ^ . 

The-first f5art will begin'with a discussion of how to organize for 

♦the proyision^of information to\arious Ijsveli of decision makers. 

Jnherent'in this is the identi^f ication of the research and evaluation 

* « *• 

department * s pr imaryjtasks, and^major const i tuents^ I n identi f^ing 
pribary^ tasks, several areas will be examified, including the mission 

'it i 

Statement of the office and each of its organizational subdivisions. 
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• The prima.ry r^^son'^d ^gtire of* the* Ofii fe' Re^^lrcb and Evaluation -is 

, ' A. " ^ ^ ^ "*'!^*%) ' ; ^ v., '^^\: ' ' •> 

'Jto service the information needs of a broad pange^^^if coosti tuencies. It 
- - • \ " . " • - * . . . » • ^ 

ircHticaJ,, therefore., to carefullV'.identi fy .the nee(ds ^nd expectations' 
of ea"ch ^rbup. Cons t i tOept *g roups to be served might' include/ but'not be 
Jimifed^tD,* the Board of Education,. 'the Super intjen^)enf\ top-level " . ' ' 

'jadministratiorr,- principals, teacher's, parents, and a v.ariety of. governmental 

» ' /• < - • . • • 

agencies^ ^ . ^ ^ ' ^ - * 

The impact , of the sou/ce^<pf fundjng 'upon-a^^prganization will ^}.so^J^e 
•considered, as will thre' interface between variously funded sub-divisVons. \ 

• • ' * * * ' r \ . ' ' 

A large urtfan school system for example., m^y have evaluation requirements 
for programs funded by' ES.EA Tit^le'i, Ti^t^ II T, Title vil, and-Title VIII. 
Im the area of early childhood education, jDrograma .may be fundejd^ through 
Head Star^, Get Set, Fol Ipw'Thrdugh, and Day Care Services grants, 
" I rregardJess of /program s imi lar }*ty^or j/omp/exity, however, eacfr^must be * 
.eval^bated according to its owri .gui'tiel ine§ and accounted for separateljy . <^ 
Jhe .a Klocat Ion of research ancj evaluation support ^ervicesVsuch as * 
programmip^g, data processing, and inatru^eot devel opment ^becomes ^ major » 
fnanageme^^roblem whi3^^rojec.t;s ace fragmented, since it is usually ^ 
'impos,^ible to provide separate, support units for each project. . 

Conc'erns related to the mayagement of large research and evajfuation ^ 
office will be developed against the backdrop of The S^chooT District of> . 



Phtladelphia's operation — - one of the nation's most comprehensive 
SGl^lrbased rese^^rch and *eva I uati on units* Pr'oblems relating to training, 
staff development, r'elease and dissemination of data, .and* the mass mpdia,'^ 



wilj 3\so be discussed* 



Scope ^\ . , • , ■ 



One of the largest pubj ic'school system| in the nation,. The School 

' D!;itricl of PJiiladelphia serves approx/Imately 275*,000 pup5Vs in pre- * * i 

"schopl ,.and^ kindergarten programs and throughout grades^ oae to twelve.. 

' « " " * ► > ♦ \* ^' . • , 'S* 

^ The District "c:mpljoys over 13,000. jteachers and has a totaf. staff of 'almos:^ 

30,000 personnel . ''.Next year*^ operating bj/dget will', exceed one h^alf of 

. ■ ■ • T ■ ' n 

a bi 1 1 ion dqUarS. ^ . i , . * . ' V ^ J '* ' 

\ ^ ^ I ' ' . . * \, 1 * ' . 

Servicing the research -and evaluation Viquirements of the School 

• r- • • ■ ■ • , ' : ' : ' ^ 

District of Phjiadelphia is the syst^m-bas^d Office. of Research and*. , • 

• ' / " ' ^ ^) ' ' { ' . ' ^ ^ ^ ^ ' 

Evaluation,* Though, som^ type* of; research unit h'as existed in Phi ladelphi^a 
, ' * \^ ■ ' '< i ' ' * * * ' • • ^ - 

for :over fifty years, ^The Office of f^esearch and 6i^a lii^t icTn (ORE) djd 
not begin to assdme its presrent scope and size until after the pass.age ^'f' 
of the Elementary and Secondary Educatpon Act of I965I Sj.nce that time, ^ 
It has grown to include five 'separate divjsjons or departments responsible 
far evaluating over ifo discrete 'projects, i n additjon to providing testing,* 
demogriphic ana lysi s , ^survey- research, data analysis, instrument develdpment, 

and, research design servicef - Approximately >25 ful I -time persons, are 

/ * * ' ' " *^ " 

employed, of ^v^hich'JS' aij;d^clerrcal or suppbrt p'er^onnel. Of the ' » 

30 professiona I research and evaluation staff merfihers about* 25 .percejnt 

possess doctorate degrees, while over three-quarlers of those. who do not, 

•are presently enrol led- in*post-mast:er^S'''3egree graduate studie^s. The , 



Office functixdns^oii^n"^nua I budget of approximaely $2,60p,000. 
Pr i'ori ties y ^ ^ • 

The goal 'of the Office of Research and Evaluation is to provide ( 
(accurate and timely info?ma4;ion to decision-makers at all Jevels. The I 
specifi^-fc priorities of thfe Office are: 



1. To produce information* for decision making •'(e.g. descriptive, 
ciomlDarative, general izabl^) arbout th^ major in^^tructional 
priorities of the School System — -Basic Skills (including 
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reading and mathematics); Early Childhood Programs'* 



Career 



. Education*; Exceptional Chiicjren; and\Al ternat ive ProgVa*ms. s 

; \2. To provide dafa and information which will.enable the School ^ • » 

District to develop, more efficient instructional and management 

* sys/ems. I ' ' ^ . - 

-^3* To develop and'present the data and informati on '^o that it meets' 

the "specif i.c need^.of the audiences for j^hich it is i'ntended and * 

SO that* the information can be used to imp.lemept' posi t ive change 

within the system. 

To provide the t^echni'cpl and non-technical support nec^ess.ary to ^\ 
/ ' ' ' . * 1 

shape the^ p ropes s of i nformati.on. gather i ng and" usSgey ,^ \ ' 

To render ser>ice on an on-going basrs, so tbat^ information^ on 



. / ' the inputs, /.processes, and 'outputs of^ a^ project or program 

V, — a^re' taken into consideration, not merely the product. / 

6^ To des i^n* i nformation S0'|hat it \s clear and meaningful/ tcf' the \ 
> " *• ^ * ■ 



'audiences to which Jt^is disseminated. 



Organizat ton ^ ' ? v 

The Executive' Director of , the Office of Research^ ant^-^valuaj^^on is ' 
.the chief admi ni strajLor. df th^voffice, and^ is responsible, for -establi l^hing 
policy, and mqpitoring the various function^ of lORE. The Office is 
divided ihto^ f ive principal uni ts : , Admi nistrat ive "and Survey ReseaV.cb 
Ser^v^ices, Iji^ractional Research and Devel^opment Services, Testing Services 
Priority Opera^tions Evt uation* Services , and Federal Evaluation Resource. 



Services'. FolTowingHn addition to a brief descriptior^ of each of-tho$e * 

units, is a series of perfo^"mance objectives against which the success of 

the or,gan.i2ationa1 unit may be appraised. The^e o|>jective.S w6re developed 

as part of the system-wide program of perfprmance appraisal^of administrators 

and supervisors;' and include, to amplify the, performance statement,, the 
J ' " * ' ^ * ^ , ^ 

reason for selection,, how to show success, and t+ie resources- needed • . 

Tables of orgaaiz^tion for each unit have been included, as an ^ ^ 

. * ft * 

\ <i ^ 

appendix.* * ' ^ ^ • * . * ^ \ 

Administrative and Survey Research Serv^ices is responsible for the 

♦development and maintenance of the pupil ^data system; provides, jn- ^ 

formational support to decision-maker^ and administrators through *socio- 

* •> ^ ' ' 1 ■ ' . * ■ 

economic and demographic analyses, pupil enrollment data^ area studies, 

» * * * , 

and«>survey research techniques; collects, tabulates, and generates data 

to meet Jihe school districtJs reporting^requirements to various federal, 

state, anc{ local agencies; and maintains arid manages the^'Super i ntendent ' s 

and District 'Superintendents* Management I nformat ion ^Center? . 

'A . • ' . ^ ^ ^ . 

Performance Objective #1 , ') . • * • 

— : — ^ 

^ To maintain, in *a timely ahd complete manner duf'ing the 1975-^76 

school year,, the Pypil ETprectory System, and to provide all ^ 

'beneficiaries of. this system with* services and products that/meet 

- their needs. « ' , 

\ .Reason for Selecting * . ;^ ' • • 

' ^ ^ * This is a mandated data system of the School District of ^ 
\ ' 

Philadelphia and. is used to -provide the resources to respond 

to various State and Federal reporting requirements, and to 
' h < ^ • * • 

r rejspbnd to the pufH 1 data needs of all clients (e.g., . . 

% , > , ' ' , / 

* Super int6r>dfe/it* 5 Office., Bqard of Education, district 



/ superintendent, indivfidual School pritici pals^'- Divi?:ior>( |- » 

' of Testing-, •Fe^ral ^ogranps, Pupil Personnelj and^lCounseh^g , ^ 

Easily C{iild^doc{*, etc.). ^ ^ " ' ^ 

, ; .How to^Show Success * * . 

Completion *of the* fall updating Cfile ini tfaJ;izatioQ) 'in 
November 1975, production of State Repor^t-d^ASt 105S( in November 
1975, production of enrollment In the Phi lade&lphia PuWi-c 
. S^chool's pdblicaiion in January.' I976, prodiictibnfpf data * ^ 
• requj red by^upri 1 Personnel and p^ounse] mg. to f^ulf i 11 the , 

State 'mandated Ahnuaf<6en5us Report irxJune 19767 file 
- availability for 'use with the 1976 Gity-wide Testing Program, . 

and interviews^ wj th afl other clients enumerated in the, 

^ 1 / 

Performance Object^ive above/. . ,^ r 

1 A 

% .Resources Needed • " , 

Director, research associate, research intern, pupil accounting 

assistant, three statistical clerks, and supportive .secretarial 

staff; per d iem* stafef 'Suri ng peak loak periods; technical support 
' ^ * ^ h i 

^ needed from Data Processing^, Division, t\ecords Managerjient ' 

Division,, and the Purchasing Division; and cooper/at iori* of 

---school principals, teacher?., "and secretaries. 

Performance. Object! ve #2 • * ^ 

, » 

To prfevi^de. the staff resources necessary to support the SchooJ 
Distrlct*s peed to have relevant, timely, and accurate dat^ 
concerning socioeconomic and demographic analyses about the pubM'c 
school population and the City of Philadelphia in general. 
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.Reason for Selecting ' r * i 

This is a capabJlity the School District of Philadelphia ^ 

must fiave .i rirbrder forNt to make well informed decisions 

and it is a specific responsibility charged to the Division 

■of Administrative and Survey Research Services. 

.How to Show Success , ^ • *^ * ^ ' ' 

* * 
Review all pervices -and produces such .Ss *short .range 

enrollment projections, desegregation s^ijxijes, attendance 

, area studies, State mandated "Long Range Developmental 

Program" Title -I eligibility, Capi tal 'Program involv^ent, 

■ ^ : 

and interviews with beneficiaries of these services. 
. r • 

".Resources Needed . ' * 

Director, assistant director, research assistant, research 

intern, pupil accounting assistant; ^two statistical clerks, 
» 

and appropriate secretarial support. 
J. 

Performance Objective #3 

To provide the School District wi^th valid and timely descriptive 
information concerning the post graduate experiences of high school 
graduates.,' the post school experiences o-f high school dropouts, 
and the. value and uses, to which standardized testing jresults are 
utilized at the individual school level, and other ad hoc surveys. 

■ ■• • ' ' ' ' 1 

Further, .to prov'hie consultative support to all Schoo.l District 
staff in survey research techniques and methodolog(. 
. .Reason for Selecting , • . . ^ * a^^- 

To continue to provide the School District with this 
valuable service which is used in a variety of ways to 



* 4 ' , ^ 

- improve the 'educational programs' In. the School Districts 

To provldfe descriptive data not available by any other 
" ^method* " ' . . 

► How to Show Sijccess - ' ^ ^ 

Review the' anndal pi^lications dealing with high school 
graduates, rfropouts, testing results, etc., and th^ value 
' that these studies have been to the School System. 
.Resources Needed ^ . * ' 

/ Director, research associ;ate for survey research, research 
assistant, research intern, two statistical clerks, secretary 
^ support, and per diem staff during the peak load periods. ' 

Performance Objective 

To maintain th,e data elements recorded i»n the Superintendent's 
Managenjent Information Center resource documents by collecting, 
editing, aggregating, and reporting all data, 'twice each year 
(June - February, July - August). . 

To insure the accuracy and completeness of all data elements 
and to investigate the feasibility of expatiding the usefulness 

• i 

of this management information 'sys^t^m-j 

•Reason for Selecting , ^ 

• • •* 

This is the responsibility given to the Dfvision of 
Administrative and Survey Research Sepvixes by the 
Superintendent . % ^ ' 

.Hov/ to Shov; Success ^ 

ilt 

beneficiaries, 



Review of the documents, materials, and irnterviews.wi th 
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•Resources Needed , \ 

•,N . • ■' - - • ^, 

Director, research assistant, research' intern, pupil 

• ' . 1, 
accounting assist/ant^ tivo. stati^^tfical clerks, graphic 

* " • - r ^ / 
• iJlustrator, qnd two- sBfcretar i es . 

, Instructional Research and fibvelbprnent Services rs. composed of- 

spepialists in the a; eas' of design and analysis, instrumerft dev.elopmervt, 

and reporljpro^ct ion. IJesearchers with special skills in data pVt>«^sing 

are also included in th*is division. Under th^ coordination of th^, 

division director, these speciaj is^'ts servQ all of tbe other divisions, 

departments^ and individuals wi,thiri ORE. The organization of these 
• . . ^ / V, ' • * • • ' 

services into , a' single unitiielps to- el imi nate duplication of services 
and schedules and provides, a standard way, resources, to theVemainder 
of ORE personnel. This groyp also functions* as a core for the propbsal 
development process. ' * - % ^^"^ 

Performance Objective #1 ^ " * 

To provide and maintain an editori^J uni t' which assists in the * 
development and production of clear and, mean4 ngf ul ORE reports* 
• .Reason for .Select! ng - . ^ - ' 

Assists in the achievement of ORE Priorities #2 and #6. 
.How. to Show Success , ^ ^ ^ 

^ Production of technical^ and' non-technical repprts, and 

non-technical abstract^, 
.Resources Needed 

Professional staff with skills', experience, and trai-ning ' 

in editing, layout proofreading, report style and production; 

Word-processing machines, personnel, and resources. 



.1 
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Performance Objec^tive ffl 

To provide and maintain a test construction and. development unit 
* . ♦ j 

which assists fn thQ.fassessment of Sfhool programs/projects. , 

• < ■» 
.Reason for Selecting 

^ Assists in .achievement of ORE Priorites #1 and #3. 

* • i • — 

f 

" * ' _ _ _ ^ 

* .How to Show Success 

New and/or prescribed ^assessment instruments for solving 
current 'measurement needs/concerns. 

♦ Resources Ne,eded ^ ^ 

Professional staff skilled in test construction, 

development, and theory. Funds to purchase resources 

\ 

materials. • . , 

Performance Objective #3 ^ . . * * * 

lo provide and maintain a flexible data managelnent pnit capable 

of processing and producing a variety of instructional information 

(e.g.y pupil test results, program comparisons). 

.Reason for Selecting • r 

Assists in the achievement of ORE Priorities #2, #^ and #5. 
.How to Show Success : ^ * ' 

Flow sheets, procedures, and forms of all operational' 

systems in current use. " , 
.Resources Needed \ 
? Professionaf staff skilled in design/ data' and" systems 

analysis; a computer facility; funds to purchase relatisd , 

resources. 



Testing Services has, the responsi'bi 1 lity for the development, 
implementation, and administration of all phases of group testing 

» H % 

programs iovolving standardized. tests such as the annual Philadelphia 
city-wide testing program. The division is a major resource in the 
local development and administration of a wi^ variety of tests for 
<use in instructional areas in' cooperatijon with the Office of 
Curriculum and Instruction. ^ * . 

The DIvFsion i^s also responsible for staff develof»ment programs • 
in coo^ration wi th^oltner offices to improve the competencies of 
teache**^, counselors*,, prirtcipals, supervisors, and other administrators 

in various phases ^of measurement, 

^ ' y * < ^ 

In fuTfilling, its role, thp division provides staff leadership 

arfd r'esource service jn the selection of/appropriate tests; management 

>^n^ adnrirYistration .of tests;^ organization and display of test data; 
^> 

interpreted Ion, use an(J dissemination of test dat?; and, in 

association with the Division of Instructional Research and 
Development, Services , Iti'^est development and scoring and subsequent 
processing aqd analysis of test data. 
•Performance Objecti ve #1 

' — ' — ^^TT - ^ ' % 

To increase the amount and quality of professional support 

service In .the interpretation and'use of city-wide testing 

< , • ^ 

> -program-data in the schools. - ^ ' * 

- , - ^ ^' ' - , 

, ^ •Reason for Selecting 

Field services are crucial components of the implementation 

t ' * ' 

\ of the city-wLde testing program in the schools. 



►How to Show Success 

Systematic survey of impact in the ^chqols; analysis of 
content and quantity of such services^, comparisons with 



^ previous yearpcr i tique of services , by school personnel* 
.Resources Needed ' - 

Additional field staff or reassignment of staff from other 
*ORE offi^ces to Division of Testing Services. 
t^erformance Objective #2 ' 

V 

Ta provide .data obtained from the city-wide testing program 
^ore* appropriatc*.l.y for* the needs of the various users^ 

St ' f 

► Reason for Selectirtg 

The major function of i:his program is to provide a variety 
of data for a variety of users ► The extent to which thi-s? 
function is achieved, will determine the success of 
this program^ * * ' * • 

..How to Shov/ Success 

Systematic survey of users; positive Responses related to 
needs of users met will indicate success. 
.Resources Needed 

Opportunities for discussions with representative members 
of the various users' groups; involvement of resources 
from other ORE offices. ^ 
Performance Objective #3 

To increase t^e amount and equality of technical and non- 
technical support involved in the" admini^stratTorTlDf the city- 
wide testing program. 



• Reason for ^Selecting . 

^ Important to the success of this program. 

%How to Show Success . ^ . 

Compare support for given program with previous 

a 

^ program, * « 

.Resources Needed 

Additional resources related to data processing 
and logistical support. - " 

Performance Objective nk 

" ^ 

To improve the clarity, accuracy and conciseness of .all 

locally-developed directi^ons, instructions^and communications j 

related to the functions of the division. 

• J- 

.Reason for Selecting 

* A significant product of this office which affects the 
f implementation of the city-wide testing program in the 

schools. ' / 

.flow to Show Success 

Systematic feed-back from school s and other users 
via survey document. 
, Resources Needed ^ 
Opportunities to obtain reactions^ suggestioQ? from . 
^ users to incorporate in future materials; involvement 
of resources from other ORE offices. 
Performance Objective nS ' 
To improve the services related to the shipment and receipt 
of testing materials for the schools in the city-wide testing 

,13- '■■-> 



program so that Niel iver ies w5 IT be both prompt dnd accurate. 

..Reason for Selecting ^ 

A significant area of service related to the city-wide 

testing program which affects administration and morale 

* in the schools. - . » 

.How to Show Success ^ \ 

Systematic feed-back from schools about meeting delivery 

target dates for this year -compared to last year via survey 

document-.-^" - — * 

.Resources Needed ' 
* 

'Sufficient lead time,, auxi 1 iary per diem, help, rapid truck 
delivery service. 
Priorij:y Operations Evaluation Services addresses itself to system 



priority areas on two dl^.j^pct Te>S(e1s>; (1) the provision of research 
assistance in priority areas""of the district superintendents, and (2) 
the evaluation of programs |n the five priority areas of the Sqhool 

? ; ■ . . . ' * 

District. The priority areas that are evaluated through POES include 
Early Childhood Education (Fol^'low Through, Get Set, Hearf StaV^t) , ' 
Career Development, Alternative Programs^, Basic Sk|;J 1^ (including ' ^ 
reading and math programs), and Special Educati^ion. Within each 



priority area fand, where appl i cable., ovi^r^^reas) POES conducts cro»s 
program analyses to attempt to i^^ntify iriajor factors^under lying 
successful pract i ces . * ^ * ' 

To develop a system of reporting which keep program 4"?^ division 

directors, including staff when appropr iate^^ current on evaluation 
information on an ongoing basiiS. i y vy 

. ■ ... 

Y ' ' ' 



Performance Objective ff\ > . 
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.Reason for Selecting ^ 
t , • 'This-:ris an attempt to bridgfe the '*guessir\g gap.** (The 

. . perloci between the time when decisions must be made and 

. - the time evaluation reports are published)- i 

^ .How to ^how Success 

Information ffow will be documented. 
.Resources Needed 

This Is largely a stiff effort. No addltronal resources 
• are' requi red. - ^ ^ 

Performance Objective §1 
^ To expand (on a demand basis) the classroom. information feedback 

system to Include*'additional. instruments so that the service 
will be more comprehensive. ■ 

.Reason for Selecting - . ■» ^ 

Availability of reliable information on pupil strengths or 
weaknesses in specific skill areas, obtained with a minimum 
. • " " of clericaTwork on the part of- the teache'r, is needed to 

help teachers plan classroom programs and assess the status 

* 

* * ' - J ,<)f current efforts. * 

. / » .How to Show Success - . <«> 

.1 ^ , ' \ 

\ ' Number of instruments' and demand for each W'l 1 1 be 

* ' , • documented^ ' 

.Resources Needed 

I ■ ^ } j Techr,4cal assistance needed, f rom Division of , I nstruceional 

' Re;search and- Oevelopment Services.- Some programming services 
' ' ' needed. Scanning forms will need to be designed and purchased. 

"^Scanning atid computer services will be\Yieed^d. 

- -15- ^ IV-'"" /ir ' 
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Performance Objective 1/3 ' "% ^ 

* • To redesign the^ management information system for the reading 

* . * ' ^ . , 

program to reTTect past experiences and present needs. ' 
• ' ♦Reason for Selecting 

Information on the success qf the reading program is needed * ^ 

* ^ \ ♦ • 

^ . to assess present strategies and plan for the future. 

• How to Show Success^ * . - - ■ 

The success. of this effort will be indicated by the aval laBi 1 i ty ' 
of the indicated Reports and the assessment of administrators 
i ^ . * of ?^he usefulness of the information* 

. Res.ources. Needed 

\ 

Programming serviced and forms design services will be needed. 
Forms" wi^ll have to be purchased. Scanning .a^compjuter 
\^ ' ^services vyi 1 1 be needed. . V ^ 4 * 

Performance Object^e 
^ ^ To design a pupi 1 . accounting system for *Day Care Services. 
♦ .Reason for Selecting 

Reports concerning Day Care Centers currently required by 
the Department of Public Welfare in Harrisburg involve a ^ ^ * 

larae amount of clerica4 vJork which can be avoided by using . ^ 
scann^^Die documents and computer process^Tng. 
.How to ShoW'.^Success _ ' * ( 



The target da^e for a functional system is September, 1975. 

%' . ' 

.Resources Needed K' ^ 

A Research Associate for Day Care Services Evaluation, has been 

assigned to the , {project- Assistance of Divisfon of Instructional • 

Research and Development Services is required. Assistance of 

/ l.O - ■ 
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Data Processing is required. Programming, Forms Design, 
Scanning** and' Computer Processing services needed. 



Performance Objecti 



/ 



Jo establish a Jongitudinal Early Childhood Education dc>ta rile. * 

* #• 

.Reason for Selecting . 

Information on the long range effects of Early Childhood 
j^rograms is needed tb assess cu|fent efforts and to plan ^ 
for the future. \ ^ - ' 

.How to SHOW Success - v , - \) 

Success will tre i'ndi-calEed' by thlspM^cal e^iJtfe^H^of ' / ' ' 
the file with* data/points common ^across Early ChTldhood 
P rog rams . ^ ^ ^ ^ \ ^ i ^ 

.Resources Needed " ^ 



ing 



\ ^ R^seardh Services "is needed. 



Prograrr^ming sreryices and the assistance^- of Data ProcBssii 
are required. The*'assistance of AdfniniS'trative i Survey 

i ' ■ 

Federal Evaluation Resource/SeftyKces conducts evaluations of pljbliic 

) ' ~ ^ ~ • ' ' ' 

scRool and non-public schopl federally funded/<:iategoricaJ projects. The ^ 

, , - , ' , * ^-^ 

^* -/ primary responsibility of FERS is on-going monitoring, data collection, and 

•<?• i . . ^'^^ ■ , • ' . • 

"'^ \ reporting the 'progress ^of Ti t i^'*!^ Title' IN", yTitle V-,' Ti.tle' VI 1 and* 



Title VIII proJects^ This department?, wkth t>ne support of the Division 



of Instructional Research and Development Services, involved in the 
establishment of design and reporting. >:The depart!/ .ft ,i5 organized to ^ 
meet the informational needs of central . aeci sfion-makers ,and the legal 
/ - ^ requirements for evaluation of federally funded projects.,! The tnajor 

roles are: ' X* . ' . ' 

-to provide information t/o centrar deci S/jon-^jakers conb^^ning the " j 
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/effectiveness of compensatory education pfrojects; 
-to pro|.-ide consulting services to central staff; 
-to er-ON\de fee !back information to^administrators of projects 

^nd schools funded under ESEA Title I, III, V. VII and VIII 
-to provide direci; services to. the Ofifice of Federal Programs 

' » • 

relative to rfeecs assessments, proposal pr^pd'rat iorr and government 
surveys . ' . " t ' ' • * 

■\ ■ - ' y' •• • 

Performance Objective ll\ 

• To provide mstructrj?n'a1 ^ahd man^ement information through systema.tic 
evaluation.of federally funded programs". ^ ' ■ . • 
- ^.Reason for Selecting i * .^'-Z' 

To provide ^assistance 'to 'dec i sign-makers at aU levels* whhfn' « 
'the school system in selecting among alternative courses^. 



\.How to Show Suc-cess ^ ^ ^ ' - ' \^ ^ 

- J- Provide periodic reports containing evaldative fiodi'nqj:/ 

' ■ " . i ^ \ 

2. Assisting the preparation of proposals and -mandated reports ' 
3- Assessing tho educational needs'of the District ' > 
.Resources Needed - , ^lju \ 

1- burrent organ t'^ati'on is sufficient' . j' 
Z. Receipt of technica 1 ^seVvices Vrpm other' Of^E units withi'n 
specific time* frames is -mandator/ i 
Perforipapce Obie ative #2 * 

^ Tp 'inc^rease the impa^ of evaluation findings by the pejiodic development 
and di^tri^ion af non^Jechnica^r reports to teachers an^ coipmunity groups 
.Reason for Selecting % ' " ' ^ 

^ - help^cilitat^ proggam develioment and modif icat'ion by, ^ , 

V prbviding^ empirical- data^ to faci*l f tat^changex ' ' /" 



.How to Show Success 

\ ■ - , • 

. * 1. Production distribution of lnte<im reports and historical 

summari'es (e.g., Title I 0\ge^) . ^ 

2. Providing consulting services in areas of efvalution to 

Parent Council S school faculties. . 

.Resources Needed 

Current organization is sufficient. 

Performance Objective r3 ; ' • ^ 

Tq provi'^e periodic reports, containing evaluation .findings to 

ispecifjed State and Federal agencies. . • ^ 

.Reason for Selecting * ^ 

I Legal «requi rement of various categorical aide programs*. 

.How to Show Success 

Production of specified reports within the prescribed time 

frames. ' " • # * 

•Resources Needed , ' " ^ ^ 

Pending*- needs will depend on the forthcomi ng guidelines 

and their implementation (P.L. 93"80) < 

C-1 ientele * * ^ ^ 

As was stated earlier in this paper^ one of the major priorities of 

the Office of Research and Evaluat^ion^ i6 responsiveness to its clientele. 

Unlike many, bther^ resear,ch and evaTluafion agencies, a school system-based 

unit such as ORE fin^^ itself serving a Jarge number of different client^l 

each with differing^ expectat ions ^s totlie products and services to be . 

V, provided them. # To get a feeling for these, the djrect.and indirect 
> r . ' 

cl)ientele served by ORE will be Ijsted. 

,v- ' . -'3- ■ ■ . 



Direct ^1 ientele^ are those individuals to whom this* Office is / ' 

directly res pons ible for the provJ^loa of certain products-or - ' 

services. They include: ^ 

.The Superintendent of'^chools -\as the chief executive of the 

School, System is in constant need of accurate, timely and 

easily communicable data. ^ • 

• , j^. ^ . . - . 

\" ^ ' ^ \ ' • ^ ' 

.Associate Superint^dents - nee^ appropriate information to aid 

them in curricujurh development and operations-oriented decision-making 

.District Su^er i^itandents - require data aggregated to reflect the** • 

schools within their districts' as well as infbrmation useful in 

dealing with each individual school community. . . 

.Princi'pals - as the i-nst?ructi.ona 1 leaders bf their buildings, Jthey 

need to know what is happen ihg in each program utilized, and in 

_every classr'oom for which they are responsible. \ 

.Teachers - for **firs^ line" personnel, fast turnaround on individual^ 

pupil progress, and ""i nformati on displayed in an understandable 

fashion is a mast. , . ' 

.Instructional Supervisors test'and evaluation results, particularly" 
in the basic skills areas, *are required. . \f . 

.Project Managers - project evaluations, geared to specific objectives, 
•enable continual process and product feedback to occur. 
,Off ice of Federal Programs and Office bf State^ Subsidies - 
'Federal and State guidelines mandate evaluations of most categorically 

funded projects. - « ^ ^ * 

.Program Directors - assessment of 'overaH • subject matier areas 
depends upon continual monitoring of progress. ' 



Indirect clientele, though they may receive almost as much service 
as di-rect cliente-le, are- not part of the formal reporting process 
of Ihe Office of Research and Evaluation* They inc'lude: 

.Board of Education - receives information from the Superintendent 

■ . 'I - 

for use in system policy planning- 
.Unions - such as the* Teachers * Federation and Administrators' * 

Association, are crucial in providing the support necessary for 

research and evaluation activities to occur. 

.Parents* Groups - such as The Home and Scho3sl Associajtion and 

The Parents Union, informally monitor the progress of the School 
* . • 1. 

System' s pupi Is. * " ^ 

.Individual Citizens - as tax payers, demand the "right to know." 
, Community Groups - at last count there were several hundred. such 
^ organizations with Interest focused on various aspects of the 

educational program as it impacts upon thei r, parti cular communitie*§ 

.City Government/- including the Mayor and City CouncH , who control 

the funding of^ the public school system.' 
.Government' Agencies - including educationa 1 ,* cl vi I rights, I'aboi^, 

and we^lfare on a local, state, and federal level;. with definite 

y 

information needs both on'a regular and ad hoc survey basis. ^ 
.Outside Research Groups - including educationbi labs and centers 
^workinfg with ORE in a number of ^iffferent cooperative endeavors 

including material development and io^trument validation. 

•Universities - prepare many of the Office's educational researchers 

\ ' 
and evaU'ator^ and originate the majority of Requests for outside ^ 

research^ sometimes on a cooperative basis. 



.Individual Researchers - generally require permission to collect* 
^data or access *to existing information. 

^ .Media - need to quickly know all results,^ trans'lated in a non- 

technica 1 fashion. 

s .Students - are the focus of the Office and of the School District 

' which it serves, and are placed here so we do not lose-sight of 

our ultimate reason for being. 

A Word About Funding 

It is not so much the lack of funds, 'as the lack of flexibility in- 

Utilizing them that creates diff icul'ties. Of the $2.6 million dollars 
e 

budget on which ORE operates, approximately $1*2 million (or ^5 percent) 
,« 

are general fund expenditures, while $1.^ million (or 55 percent) are 
obtained from Federal funds. 

. Almost half of the general fund expenditures ^re utilized for the 

city-wide testing program. The remainder of the monies are ear-marked 

■ - ft 
for technical support personnel, a few scattered general fund pViority 

projects, and office administration. Virtually all Federal evaluation 

funds are designated for the evaluation of specific Titled projects. 

What is missing is an individual to function as an Evaluation Auditor, 

. aijd as such, to be responsible for evaluating the effectiveness of th^ 
# 

Office of Research and EvaluatioT\ and for making recoirenendat ions 

/' 

regarding utili^a.tion of personnel and methodology for evaluation of , both 

^ - I ^ 

Ijederally-funded agd operating budget programs. ' Since such information 

might be of a highly sensitive nature, in- that the present activities 

/ 

s 



and structures are to be scrutinized and judged, the Evaluation Auditor 
-e 

) 



would reporj/)di recti y to the Executive Director of the Office. Such 
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an organization would help to assure the independence and objectivity 
of the position. 

In addition, a means needs to be developed whereby full time professional 
staff can be made available to work on special projects, often ad hoc in 
nature, regardless of their source of funding. Unless this tan be 
accomplished, the level of utility which flexibility, of operations 
engenders, may never be reached. 
Concerns and Solutions 

Managing an office as large and diverse as the Office of Research 
and Evaluation of the School District of Philadelphia can and does 
create a variety of different problems. One could easily spend an 
entire presentation dealing only with them. Instead, however, it 
might be more productive to focus on ways in which major problem areas 
are being dealt with. » 

In a' rapidly expanding office, particularly one which deals with a 
subject as embryonic as educational evaluation, there will be a 'tremendous 
need for staff development and training. To meet this need ORE has 
adopted a multi-faceted approach to staff development. New research 
assistants and research interns are provided with a two-day intensive 
overview of evaluation and its place in the public school setting. 
These workshops are taught by ORE senior staff members who are in a^ 
position to not only provide technical information, but are very 
familiar with the day-to-day activities of the office and can readily 
translate theory into practice. In addition, a number of ongoing 
series of more technical classes are taught on a rotating basis and 
are open to anyone wishing to gain new knowledge or update past under- 
standings in such areas as statistics, research design, and computer 
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applications. THfese, too, are taught by In-house Knstruttors, many 

-* • * " , 

■of whom also teach an occasional evening class at a locaJ unJversjty. 

The- entire sta/f of ORE is encotraged to participate In monthly 

coffee 'hours and lunch time seminars. The coffee hours provide an 

hour break during one morning a mop^th and enable staff members from 

" y i — • ■ . 

•^^.^'different projects to informal ly catch up -on what Is haf)pening. 

Brief, noy-iechnicaj presentations ace often made during these sessions. 
Lunch time seniljnars are more technically oriented and are devoted to 
^formal presentation and^ discussion of a topic related to the activities 

of the offi^ce. At least one two-day st*aff trai nl ng* session per year is 

\ f' ; * ^ " . . " - 

also held'<on an off-site location. The major foci .of'such sessions, 

to date,, have b^en planning and communication. 

Another problem with which ORE has been^ forced to deal r4lates to . ' 

excessive ddmands fronv outside agencies,- organizations, Snd Individual^ 

for permission to dp research within the School District. The 'response 

to this problem has been to develop system wide guidelines relating 

• / * 

to research policies,* as well as c rersearch review committee responsible 

fqr reviewing, and. ultimately rejecting or facilitating, outside research 

.projects within the "confines of the" Schobi District. 

Current federal and state- legislation has created still another » 
. \ 3 

dilemma in terms of the often conflicting regulations governl ng. fu 1 1 

, ■ "I 

disclosure Qf information v^. privacy of data. Unable to wait for -m 
ultimate clarification of such conflicts', ORE has'^ developed a policy 
on the release of research data which, while meeting the'intent of the 
law, still enables research and evaluation Activities to be carried on. ^ 



\The identification of client nfeeds has posed a problem simply 

*'bfecause- it is *ap area which was not dealt with in the past, in 

- responding to i.e. planning -sessions have been scheduled with project 

managers, program director,s and members of thk superintendent's cabinet 

. In.adaiti-on. 'al l of the" ORE^unit heads \rave'l as a group to each of the 

city's eight ciis-trlct offices on an, annua 1 basis for the purpose of 

meeting with the distri'ct superintendent and members of his planning 
team. ' * * - . . 

. Fi^lly, better communication and dissemination has proved to be a 
major concern."' in deal Tng with it, ORE has adopted the follov/ing 
approaches: ' 

.Media seminars to pxplain test and evaluation results. 

i 

, -.A program to improve -information flow to the media' 
^ * »^ . ■ - . 

/Superintendent's cabinet b>iefing sessions on major findings 
j • and results. -> » 

.Prese-ntations. of test r-esults and certa-in program evaluations 
on .pufilicly televised -Board of . Education meetings, 

.A new publications policy .which requires an, abstract and summary 
to precede every techn ical repbrt and encourages the use of 
compani.on, poptilarized versions to many , ORE products^ 

.The development^ of^Manageme.nt Information Center materials 
-updated on -a, -semi -annua r b3S is, to provide the Board of Edu<?ation 
X .and SMpenntendent v^ith baseline data of several dozen key system 
\ variables:-, ,- , ' ' 

■?J^^i?fr"-^! "u""'^'^ oOpublic interest groups (ejtg., Title I 
Par.fent^ Council, Home S School Association, etc.) to explain 
testPftg-and evaluati-on 'rfesults, and to answer questions concerning 
(the Office of Resea^rch and Evaluation 'program. . 

Signs \of Success J. " • - ' ■ ' 

It y/gi b'e.manj years indeed- before research and evaluation, activitites 
are,recogn-j,2ed a^nd ,y.t i l.ized as an important factor in the planning and 
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daily operation of school systems* It will very likely be even longer 

before all but the larger or more progressive districts devote more 

than lip service to this area, ' . - 

The road to acceptance of the Office of Research ''and Evaluation in 

Philadelphia, though becoming smoother, is still fraught with many an 

obstacle. Nevertheless, four recent developments are viewed as Signs 

of future success, 

-ORE is now represented, for the first time, on the Superintendent's 
Cabinet. 

-Board of Education members are availing themselves of ORE-produced 
data, with increasing frequency. 

-Top level administration has adopted a policy requiring that all 
new programs have included in them an evaluaftion component. 

-ORE-produced data has been accepted as evidence in a number of 
recent court suits. 

m 

If the Office of Research and Evaluation is able to continue to provide 
information which is viewed as useful by its varied clientele, it cannot 
help but eventually win acceptance and support. 



PS 
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